Building Slow Development Systems (On Purpose)
I had an interesting discussion last week with an Oracle DBA that I have a lot of respect for. He surprised me by telling me that he thought development systems should intentionally be built to be slow. That way, the developers would be forced to write efficient code. He was not just stating a philosophical position either. He had actually built some databases with 32M buffer caches (the one I mentioned in this previous post on Buffer Cache Extremes was one of his). He went so far as to say that he thought that developers should be using 286’s for their development systems.
I have argued for several years that development systems should match production as closely as possible. But that conversation really got me thinking. Maybe intentionally slowing down development systems is not such a bad idea…
So what do you think?
Several problems with that.
Firstly you get the developers building and tuning to THEIR bottlenecks. For example, they might ‘save’ memory or disk on their machine at the expense of re-querying data over a network. If the end user is on a slow network, then the optimization will actually work against them.
Secondly, developer productivity is slowed. They may have to keep shutting down the IDE to run a test….so they test less.
Thirdly, the developers get annoyed at having such crummy equipment so they leave.
Fourthly, if you get a performance issue on the production box, there’s a chance you won’t be able to reproduce them on the developer’s box. So fixing them becomes a matter of try something, migrate to test (or even prod if you have slowed test too), test, if it doesn’t help, goto 1.
Gary,
Well don’t hold back! Tell us how do you really feel!
Seriously though, I agree with you (I think). To your point number 1, I definitely think you can influence what the developers do by how the system is built (which I think was my friend’s point). I hadn’t really given it any thought before, but making the buffer cache really small basically turns all the logical i/o into physical i/o, therefore “encouraging” the developers to minimize the logical i/o’s. Maybe not such a bad thing.
Points 2-4 on the other hand could be serious draw backs.
Kerry
And, at a place I used to work, where the dev systems were made from older, slower kit, any performance limitations in the development phase were glossed-over with “oh, well, it’s always faster when it goes onto the production kit”. So there was no development incentive to tune early.
I agree with Peter, imho If there is an excuse for the dodgy code that excuse will be used sooner or later.
The developer who does not care about performance will not improve his code, he will just complain about the environment.
But it is an interesting approach!
[…] say you got a VM server or slow development server and most of the time users are already crying for slow performance on DEV system when they compare […]
Hmmm, I thought this one might generate a little discussion.
Peter and Coskan, I don’t think my friend told the developers that he had intentionally reduced performance of Dev by making the buffer cache so small. So I don’t think they would have that info readily available as an excuse ;). But I do get your point. If the developers feel beat down, eventually they will just quit trying (or start blaming the Dev system).
I have to say that my first reaction was that this approach was just wrong. But for some reason, my brain keeps replaying the conversation. So I’m not sure what that means, but it is interesting.
Kerry
“I don’t think my friend told the developers that he had intentionally reduced performance of Dev by making the buffer cache so small.”
But if the developers know how to performance tune, they should be able to see that the physical IOs are a high proportion of logical IO (oh no, buffer cache hit ratio strikes again). Or maybe, by hit and miss, they find out that caching data in PL/SQL variables/tables makes it go faster. Which is great on a single user development environment. Not so good in Prod when you have a few hundred sessions with inflated PGA all caching the same values. Or maybe they push everything to a Java layer, whinge about Oracle being too slow and how they should move to mySQL/SQL Server/Postgres.
Sorry, but the DBA artificially hampering the developers strikes me as playing politics rather than an intention to deliver quality solutions to the business.
Hey, you left out my smiley, winking face ;).
I agree with you that negative side affects could occur, maybe more frequently than the desired results. And I agree with you, manipulation is a four syllable word. But on the other hand, most development systems I’ve seen are less capable than the production systems they support (but usually not in a methodical way). By that I mean that they are usually made up of left over parts, older production systems for example that have been replaced. And they seem to do the job. So I’m not sure that a more methodical approach to slowing down a specific operation may not yield desirable results. I really haven’t made up mind yet, but I find the idea quite interesting.
By the way, I posted the same basic info on the Oracle-l list and have had several responses. Most are in the “YOU’VE GOT TO BE KIDDING!” camp, but a few have said that they do intentionally set up development system to be slower.
By the way number 2, my friend didn’t put it nearly as harshly as I did. He said something to the effect that if the code performed well on the less capable development system it had a much better chance of performing well when it was moved to production. His position may well be motivated by the expertise of the development group he is working with. (I doubt seriously that they ever heard of a buffer cache hit ratio) But like I said, I’m still not convinced.
Kerry
Just FYI. I requested feedback on the idea on the oracle-l mailing list a couple of days ago and there were several responses. It was more negative than positive – although there were a few that were in favor of this approach. You can see the related messages here: Building Slow Development Systems on Purpose
I found it amusing that at the end the messages started to degenerate into how to make things run slow. Stephane Faroult recommended setting a small SDU to slow down SQL*Net traffic and putting redo logs on slow/busy disks to slow down commits. Cary Milsap mentioned setting protocol=tcp (as opposed to ipc) to exaggerate network latency. Yong Huang recommended setting commit_write = ‘immediate,wait’ to slow down commits. That list is usually all about how to speed things up. I thought it was an ironic twist.
Kerry
In the year 1999, we had to develop one of the first 3-tier applications using oracle forms. The development was great, test even better but in the productions things were not so good. Users were complaining for slow responses. After two painful days, we discovered that 50% of the users were in the WAN environment and another 50% in LAN.
I changed the development environment using simulated WAN lines (within the LAN) and I faced exactly the same pain.
A lot of useless sqlnet round trips because of bad buffer size, inappropriate settings at TCP level, some troubleshooting at listener level and of course the production data didnt match test data.
My conclusion is the following
1. Use slow hardware but the same operating system and database version
2. Generate a lot of data similar to production to check queries, statistics, histograms etc
3. Use similar network as in production
In this way, the developers from the beginning they will try the best by themseves
You have an excellent blog
Thank you
Kostas Hairopoulos
That’s a good perspective. I agree that may be making it slow will result in writing an efficiet code!